<html>
	<head>
        <title>An Attempt at Demystifying Bayesian Deep Learning</title>
        <!-- Head.JS -->
        <script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0-beta.2/js/bootstrap.min.js" integrity="sha384-alpBpkh1PFOepccYVYDB4do5UnbKysX5WZXm3XxPqe5iKTfUKjNkCk9SaVuEZflJ" crossorigin="anonymous"></script>
        <script type="text/javascript" src="lib/js/head.min.js"></script>
        <link rel="stylesheet" href="lib/css/atom-one-light.css">
		<link rel="stylesheet" href="css/reveal.css">
		<link rel="stylesheet" href="css/theme/simple.css">
        <link rel="stylesheet" href="css/reveal-overrides.css">
    </head>
	<body>
		<div class="reveal">
			<div class="slides">
				<section id="Introduction Slide">
                    <h2>An Attempt At Demystifying Bayesian Deep Learning</h2>
                    <p>Eric J. Ma</p>
                    <p><i class="fa fa-twitter" aria-hidden="true"></i><i class="fa fa-github"></i> ericmjl</p>
                    <p><i>PyData NYC 2017</i></p>
                </section>
                <section id="follow-along!">
                    <h2>Follow along!</h2>
                    <section>
                        <h3>On your phone</h3>
                        <img src="images/qrcode.svg" alt="" width=30%>
                    </section>
                    <section>
                        <h3>On your laptop</h3>
                        <a href="https://ericmjl.github.io/bayesian-deep-learning-demystified">https://ericmjl.github.io/bayesian-deep-learning-demystified</a>
                    </section>
                </section>
				<section id="Outline and Take-Home Points" data-transition="zoom">
                    <section>
                        <blockquote align="center" class="twitter-tweet" data-lang="en">
                            <p lang="en" dir="ltr">The Hype of Deep Learning:
                                <br>1. Write a post with ML, AI or GAN in the title.
                                <br>2. post appears at the top of hackernews (despite your best efforts)
                                <br>3. HN drives tens of thousands of clicks
                                <br>4. &quot;what&#39;s with all the maths? show me pretty pics&quot;
                                <br>5. &lt;=1% stay for longer than a minute
                            </p>&mdash; Ferenc Huszár (@fhuszar)
                            <a href="https://twitter.com/fhuszar/status/933682386713350145?ref_src=twsrc%5Etfw">November 23, 2017</a>
                        </blockquote>
                        <p class="fragment"><b>I am out to solve Point 4.</b></p>
                        <script async src="https://platform.twitter.com/widgets.js" charset="utf-8">
                        </script>
                    </section>
                    <section>
                        <h3>The Obligatory Neon Bayes Rule Sign</h3>
                        <img src="images/bayes-rule.jpg" alt="">
                    </section>
                    <section>
                        <h2>My (Modest) Goals</h2>
                        <p>
                            <ul>
                                <li class="fragment">Demystify Deep Learning</li>
                                <li class="fragment">Demystify Bayesian Deep Learning</li>
                            </ul>
                        </p>
                        <p class="fragment">Basically, <i>explain the intuition clearly with minimal jargon</i>.</p>
                    </section>
                    <section data-transition="fade">
                        <h2>Take-Home Point 1</h2>
                        <p>
                            Deep Learning is nothing more than <b>compositions of functions on matrices</b>.
                        </p>
                        <img src="images/deepnet_regressor-matrices.png" alt="" width=40%>
                    </section>
                    <section data-transition="fade">
                        <h2>Take-Home Point 2</h2>
                        <p>Bayesian deep learning is grounded on <b>learning a probability distribution for each parameter</b>.</p>
                        <img src="images/deepnet_regressor-matrices-bayesian.png" alt="" width=40%>
                    </section>
                </section>
                <section id="Assumptions">
                    <h2>Outline</h2>
                    <p>
                        <ol>
                            <li class="fragment">Linear Regression 3 Ways</li>
                            <li class="fragment">Logistic Regression 3 Ways</li>
                            <li class="fragment">Deep Nets 3 Ways</li>
                            <li class="fragment">Going Bayesian</li>
                            <li class="fragment">Example Neural Network with PyMC3</li>
                        </ol>
                    </p>
                </section>
                <section id="Linear Regression" data-transition="zoom">
                    <section id="linreg-title">
                        <h2>Linear Regression</h2>
                    </section>
                    <section id="linreg-equation-form">
                        <h3>Function</h3>
                        <img src="images/linreg.tex.png" alt="" width=60%>
                    </section>
                    <section id="linreg-matrix-diagram">
                        <h3>Matrices</h3>
                        <img src="images/linreg-matrices.png" alt="" width=80%>
                    </section>
                    <section id="linreg-neural diagram">
                        <h3>Neural Diagram</h3>
                        <img src="images/linreg-neural.png" alt="" width=60%>
                    </section>
                    <section id="linreg-side-by-side">
                        <h3>LinReg 3 Ways</h3>
                        <img src="images/linreg-all.png" alt="" width=90%>
                    </section>
                </section>
                <section id="Logistic Regression" data-transition="zoom">
                    <section>
                        <h2>Logistic Regression</h2>
                    </section>
                    <section id="logreg-equation-form">
                        <h3>Function</h3>
                        <img src="images/logreg.tex.png" alt="" width=40%>
                    </section>
                    <section id="logreg-matrix-diagram">
                        <h3>Matrices</h3>
                        <img src="images/logreg-matrices.png" alt="" width=70%>
                    </section>
                    <section id="logreg-neural-diagram">
                        <h3>Neural Diagram</h3>
                        <img src="images/logreg-neural.png" alt="" width=70%>
                    </section>
                    <section id="logreg-side-by-side">
                        <h3>LogReg 3 Ways</h3>
                        <img src="images/logreg-all.png" alt="" width=90%>
                    </section>
                </section>
                <section id="Deep Neural Networks" data-transition="zoom">
                    <section>
                        <h2>Deep Neural Networks</h2>
                    </section>
                    <section id="dnn-equation-form">
                        <h3>Function</h3>
                        <img src="images/deepnet_regressor.tex.png" alt="" width=50%>
                    </section>
                    <section id="dnn-matrix-diagram">
                        <h3>Matrices</h3>
                        <img src="images/deepnet_regressor-matrices.png" alt="" width=50%>
                    </section>
                    <section id="dnn-neural-diagram">
                        <h3>Neural Diagram</h3>
                        <img src="images/deepnet_regressor-neural.png" alt="" width=90%>
                    </section>
                    <section id="dnn-side-by-side">
                        <h3>DeepNets 3 Ways</h3>
                        <img src="images/deepnet_regressor-all.png" alt="" width=90%>
                    </section>
                </section>
                <section>
                    <section>
                        <h2>Going Bayesian</h2>
                    </section>
                    <section>
                        <p>Key Idea: Learn probability density over parameter space.</p>
                    </section>
                </section>

                <section id="Bayesian Linear Regression" data-transition="slide">
                    <section id="bayesian linreg title" data-transition="fade">
                        <h2>Bayesian Linear Regression</h2>
                    </section>
                    <section id="intuition">
                        <h3>Intuition</h3>
                        <table>
                            <tr>
                                <td align="center" class="fragment"><img src="images/linreg-non-bayesian.png" alt="" width=60%></td>
                                <td align="center" class="fragment"><img src="images/linreg-bayesian.png" alt="" width=60%></td>
                            </tr>
                        </table>
                    </section>
                    <section data-transition="fade">
                        <h3>From this...</h3>
                        <img src="images/linreg-matrices.png" alt="" width=80%>
                    </section>
                    <section data-transition="fade">
                        <h3>..to this</h3>
                        <img src="images/linreg-matrices-bayesian.png" alt="" width=80%s>
                    </section>
                </section>
                <section id="Bayesian Logistic Regression" data-transition="slide">
                    <section>
                        <h2>Bayesian Logistic Regression</h2>
                    </section>
                    <section data-transition="fade">
                        <h3>From this...</h3>
                        <img src="images/logreg-matrices.png" alt="" width=70%>
                    </section>
                    <section data-transition="fade">
                        <h3>...to this</h3>
                        <img src="images/logreg-matrices-bayesian.png" alt="" width=70%>
                    </section>
                </section>
                <section id="Bayesian DeepNet Regressor" data-transition="slide">
                    <section>
                        <h2>Bayesian Deep Nets</h2>
                    </section>
                    <section data-transition="fade">
                        <h3>From this...</h3>
                        <img src="images/deepnet_regressor-matrices.png" alt="" width="60%">
                    </section>
                    <section data-transition="fade">
                        <h3>...to this</h3>
                        <img src="images/deepnet_regressor-matrices-bayesian.png" alt="" width="60%">
                    </section>
                </section>
                <section id="Ever-Expanding Cheat Sheet" data-transition="zoom">
                    <h2>Cheat Sheet</h2>
                    <img src="images/infographic.png" alt="" width=90%>
                </section>
                <section id="pymc3-intro">
                    <section>
                        <img src="images/pymc3.svg" alt="" width=30%>
                        <p>Probabilistic Programming in Python. Provides:</p>
                        <ul>
                            <li class="fragment">statistical distributions</li>
                            <li class="fragment">sampling algorithms</li>
                            <li class="fragment">syntax</li>
                        </ul>
                    </section>
                </section>
                <section id="Bayesian Deep Net Multiclass Classification">
                    <h2>Predict Forest Cover Type</h2>
                    <section id="examples-problem-overview">
                        <h3>Problem Overview</h3>
                        <p>
                            <ul>
                                <li class="fragment">UCI ML Repository: <a href="https://archive.ics.uci.edu/ml/datasets/covertype">Covertype Dataset</a></li>
                                <li class="fragment"><b>Input</b>: 66 cartographic variables</li>
                                <li class="fragment"><b>Output</b>: one of 7 forest cover types</li>

                            </ul>
                        </p>
                    </section>
                    <section id="examples-intro">
                        <h3>Network Architecture</h3>
                        <img src="images/forest-cover.png" alt="" width=60%>
                    </section>
                    <section id="examples-code">
                        <pre>
                            <code data-trim data-noescape class="py">
                            import theano.tensor as tt  # pymc devs are discussing new backends
                            import pymc3 as pm

                            n_hidden = 20

                            with pm.Model() as nn_model:
                                # Input -> Layer 1
                                weights_1 = pm.Normal('w_1', mu=0, sd=1,
                                                      shape=(ann_input.shape[1], n_hidden),
                                                      testval=init_1)
                                acts_1 = pm.Deterministic('activations_1',
                                                          tt.tanh(tt.dot(ann_input, weights_1)))

                                # Layer 1 -> Layer 2
                                weights_2 = pm.Normal('w_2', mu=0, sd=1,
                                                      shape=(n_hidden, n_hidden),
                                                      testval=init_2)
                                acts_2 = pm.Deterministic('activations_2',
                                                          tt.tanh(tt.dot(acts_1, weights_2)))

                                # Layer 2 -> Output Layer
                                weights_out = pm.Normal('w_out', mu=0, sd=1,
                                                        shape=(n_hidden, ann_output.shape[1]),
                                                        testval=init_out)
                                acts_out = pm.Deterministic('activations_out',
                                                            tt.nnet.softmax(tt.dot(acts_2, weights_out)))  # noqa

                                # Define likelihood
                                out = pm.Multinomial('likelihood', n=1, p=acts_out,
                                                     observed=ann_output)

                            with nn_model:
                                s = theano.shared(pm.floatX(1.1))
                                inference = pm.ADVI(cost_part_grad_scale=s)  # approximate inference done using ADVI
                                approx = pm.fit(100000, method=inference)
                                trace = approx.sample(5000)
                            </code>
                        </pre>
                    </section>
                    <section id="layer-1-weights" data-transition="fade">
                        <h3>1st Layer Weights</h3>
                        <img src="images/layer1-weights.png" alt="" width=100%>
                    </section>
                    <section id="layer-2-weights" data-transition="fade">
                        <h3>2nd Layer Weights</h3>
                        <img src="images/layer2-weights.png" alt="" width=100%>
                    </section>
                    <section id="output-weights" data-transition="fade">
                        <h3>Output Weights</h3>
                        <img src="images/layer3-weights.png" alt="" width=100%>
                    </section>
					<section id="class-predictions" data-transition="fade">
	                   <h3>Class Predictions</h3>
                       <img src="images/class_predictions.png" alt="" width=50%>
                       <p><i>"point estimate"</i></p>
					</section>
                    <section id="class-probabilities" data-transition="fade">
                        <h3>Class Probabilities</h3>
                        <img src="images/class_probabilities.png" alt="" width=50%>
                        <p><i>"probabilistic estimate"</i></p>
                    </section>
                    <section id="class-uncertainties" data-transition="fade">
                        <h3>Class Uncertainties</h3>
                        <img src="images/class_uncertainties.png" alt="" width=50%>
                        <p><i>"with uncertainties!"</i></p>
                    </section>
                </section>
                <section id="recap">
                    <section id="recap-1" data-transition="fade">
                        <h2>Take-Home Point 1</h2>
                        <p>
                            Deep Learning is nothing more than <b>compositions of functions on matrices</b>.
                        </p>
                        <img src="images/deepnet_regressor-matrices.png" alt="" width=40%>
                    </section>
                    <section id="recap-2" data-transition="fade">
                        <h2>Take-Home Point 2</h2>
                        <p>Bayesian deep learning is grounded on <b>learning a probability distribution for each parameter</b>.</p>
                        <img src="images/deepnet_regressor-matrices-bayesian.png" alt="" width=40%>
                    </section>
                </section>
                <section id="resources-all">
                    <section id="resources">
                        <h2>Resources</h2>
                        <ul>
                            <li><a href="http://docs.pymc.io/index.html">PyMC3 docs</a></li>
                            <li><a href="https://github.com/ericmjl/bayesian-analysis-recipes">Bayesian Analysis Recipes</a></li>
                        </ul>
                    </section>
                    <section id="teachers">
                        <h2>Teachers</h2>
                        <p>
                            <ul>
                                <li>David Duvenaud</li>
                                <li>Michelle Fullwood</li>
                                <li>Thomas Wiecki</li>
                            </ul>
                        </p>
                    </section>
                    <section id="people-to-read">
                        <h2>People to Follow</h2>
                        <p>
                            <ul>
                                <li>David MacKay</li>
                                <li>Yarin Gal</li>
                            </ul>
                        </p>
                    </section>
                </section>
                <section id="Thanks!">
                    <h1>Thank you!</h1>
                </section>
			</div>
            <div class="footer">
                <p>Source: <i class="fa fa-github" aria-hidden="true"></i> ericmjl/bayesian-deep-learning-demystified</p>
            </div>
		</div>
        <script type="text/javascript" src="js/reveal.js"></script>
		<script>
			Reveal.initialize({
                width: "100%",
                height: "100%",
                margin: 0,
                minScale: 1,
                maxScale: 1,
                // Push each slide change to the browser history
                history: true,
                // Display the page number of the current slide
                slideNumber: true,
                dependencies: [
                    // Syntax highlight for <code> elements
                    { src: 'plugin/highlight/highlight.js', async: true, callback: function() { hljs.initHighlightingOnLoad(); } },
                    // MathJax
                    { src: 'plugin/math/math.js', async: true }
                ]
                });
		</script>
        <script src="https://use.fontawesome.com/8f66f558d2.js"></script>
	</body>
</html>
